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We present a search for flavor changing neutral currents via quark-gluon couplings in a sample 
of single top quark final states corresponding to 2.3 fb^^ of integrated luminosity collected with 
the DO detector at the Fermilab Tevatron Collider. We select events containing a single top quark 
candidates with an additional jet, and obtain separation between signal and background using 
Bayesian neural networks. We find consistency between background expectation and observed data, 
and set limits on flavor changing neutral current gluon couplings of the top quark to up quarks 
(tgu) and charm quarks (tgc). The cross section limits at the 95% C.L. are atgu < 0.20 pb and 
o-tgc < 0.27 pb. These correspond to limits on the top quark decay branching fractions of B{t 
gu) < 2.0 X lO""* and B{t ^ gc) < 3.9 x 10"^ 
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quarks was reported in 2009 [H, [ll- Those measurements, 
as well as the evidence for i-channel single top quark 
production focus on standard model (SM) production 
of single top quarks. The single top quark final state 
is sensitive to several models of physics beyond the 
SM in particular those in which flavor changing 
neutral current (FCNC) couplings between a gluon, a 
top quark, and up or charm quarks may be large. 
Examples include models with multiple Higgs doublets 



such as supersymmetry [5|-|7[, those with new dynamical 
interactions of the top quark or models in 



(a) 



(b) 



or 



which the top quark is a composite object 
has a soliton structure [l2l . 13 1. In principle, such 



Hi 



interactions can also be produced through SM higher- 
order radiative corrections; however, their effects are 
too small to be observed |5|. Stringent limits exist for 
FCNC top quark couplings to photons and Z bosons 
from studies of production and decay of top quarks 
\v\ . The first limits on gluon FCNC couplings to 
the top quark were obtained in a DO analysis based 
on 0.23 fb-i of integrated luminosity The CDF 

Collaboration searched for FCNC production of single 
top quarks, without extra jets, through gluon-quark 
interactions using a dataset corresponding to 

2.2 fb~^ of integrated luminosity [2lj. The limits on 
the FCNC couplings are «;tg„/A < 0.018 TeV^^ and 
Ktgc/A < 0.069 TeV~^, where A is the scale of the 
new interactions which generate these couplings (of order 
1 TeV). 

The FCNC coupling of a gluon to a top quark and a 
light quark results in either s-channel production and 
decay [Fig. [1] (a) and (d)] or in t-channcl exchange 
[Fig. [T] (b) and (c)] of a virtual particle. The largest 
contribution to the production cross section (83% for tgu 
and 66% for tgc) is from the diagram in Fig. [1] (a). The 
final state in each case contains a top quark and a light 
quark or gluon, a topology similar to SM t-channel single 
top quark production. We do not consider the single top 
quark final state without extra jets that was explored by 
the CDF Collaboration [2l| due to its different final state 
topology and significantly smaller signal event yield for 
a given coupling. 

The FCNC couplings can be parametrized in a model- 
independent way. using an effective Lagrangian [22. 23| 
of the form: 



FCNC - ~J^3s.t'^'^ -^rLr^,.7 



(1) 



where / = m or c, with u, c and t representing the 
quark fields; Kfgf defines the strength of the tgu or tgc 
couplings; gg and A" are the strong coupling constant and 
color matrices; a'^'^ and G°j^ are the Dirac tensor and 
the gauge field tensor of the gluon. The FCNC single 
top quark production cross section depends therefore 
quadratically on the factor Ktgf/A. For a coupling of 
Ktgf/A = 0.015 TeV"\ the next-to leading order (NLO) 
cross sections at a top quark mass of 170 GeV are 
cTt^^o ^ 0.29 pb and cr^LO = o.020 pb The top 

quark decay branching fraction to a gluon and any quark 
also depends quadratically on the factor Ktgf / A [25| , but 
this branching fraction is iiegligible for coupling factors 
considered in this analysis |23| . 

We search for FCNC production of single top quarks 
in association with a quark or gluon, where the top quark 
decays to a, W boson and a b quark, and the W boson 





(c) 




FIG. 1: Leading order Feynman diagrams for FCNC gluon 
coupling between an up or a charm quark and a top quark, (a) 
and (d) show two s-channel diagrams for the tg final state and 
the tq final state and (b) and (c) are two f-channel diagrams 
for the tq final state. The circles indicate the effective FCNC 
coupling, possible at either of the two vertices in (a) and (c), 
for which the amplitudes are properly summed. 



subsequently decays to a lepton (electron or muon) and 
a neutrino. The main backgrounds to this final state 
arc from M^4-jets production, including W + c-quarks 
and W + 6-quarks, with smaller contributions from ti, 
SM single top quarks [tb + tqb), as well as multijets, 
dibosons, and Z+jets production. We base the analysis 
on the dataset and event selection from the single top 
quark production observation Letter [ij, using 2.3 fb~^ of 
integrated luminosity collected with the DO detector [2^ 
at the Fermilab Tcvatron Collider. 

The detector has a central tracking system, consisting 
of a silicon microstrip tracker and a central fiber 
tracker, both located within a 1.9 T superconducting 
solenoidal magnet, with designs optimized for tracking 
and vertexing at pseudorapidities \ri\ < 3 and \r]\ < 
2.5, respectively 27-2^. A liquid-argon and uranium 
calorimeter has a central section covering pseudora- 
pidities \r]\ up to w 1.1, and two end calorimeters that 

extend coverage to [ttJ « 4.2, with all three housed 

in separate cryostats [30[. An outer muon system, at 
1 77! < 2, consists of a layer of tracking detectors and 
scintillation trigger counters in front of 1.8 T toroids, 
followed by two similar layers after the toroids [sij . 

We select events containing a lepton, missing 
transverse energy {^t), and two to four jets with 
transverse momentum px > 15 GeV and \r]\ < 3.4 
(allowing for jets from gluon radiation), with the leading 
(highest pt) jet additionally satisfying px > 25 GeV [s^ . 
We require 20 < < 200 GeV for events with two jets 
and 25 < < 200 GeV for events with three or four 
jets. Events must contain only one isolated electron with 
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Pt > 15 GeV and |7?| < 1.1 (pr > 20 GeV for three- or 
four-jet events), or one isolated muon withp^ > 15 GeV 
and I77I < 2.0. The multijets background, where a jet 
is misidentified as an isolated lepton, is kept to approxi- 
mately 5% of the total background by requiring the scalar 
sum of all transverse energies, (lepton, alljets), 
to be greater than 110 to 160 GeV, depending on the 
lepton flavor and jet multiplicity, and by requiring that 
the is not colinear with the axes of the lepton or 
the leading jet in the transverse plane. To enhance the 
fraction of top quark events, one of the jets is required to 
be identified as originating from b quark fragmentation 
through a neural network (NN) 6-tagging algorithm [33| . 
To partially reject background from W + bb, ti, and SM 
single top quark events, each event is required to contain 
only one ^-tagged jet (vetoing double-tagged events) , in 
contrast to SM single top quark analyses where double- 
tagged events are also considered. 

We model the FCNC signals and SM single top 
quark background with the SingleTop Monte Carlo 
(MC) generator 3i|, using CTEQ6M parton distribution 
functions [sMISgl. The ALPGEN leading-order MC event 
generator [37|, interfaced to pythia for showering and 
hadronization [s^, is used to model ti, W+jets, and 
Z+jets background, while pythia is used to model 
diboson (WW, WZ and ZZ) production. We set the 
mass of the top quark to 170 GeV, and use the CTEQ6L1 
parton distribution functions [sl,!!^. We use GEANT [s^I 
to simulate the response of the DO detector to MG events. 
To model the effects of multiple interactions and detector 
noise, data from random pp crossings are overlaid on 
MC events. The SM single top quark, tt, diboson and 
Z+jcts backgrounds are normalized to their predicted 
cross sections The M^-|-jets background normal- 

ization and jet angular distributions are obtained from 
data samples without ^-tagging requirements, and its 
flavor composition is determined from data samples 
with different numbers of 6-tagged jets. We model 
the background from multijets production using data 
containing lepton candidates that fail one of the lepton 
identification requirements, but otherwise resemble the 
signal events. In the muon channel, where a secondary 
muon in a jet is misidentified as an isolated muon, this 
is accomplished by reversing the tight isolation criterion, 
whereas in the electron channel, where a jet is misiden- 
tified as an electron, we reverse the tight electron identi- 
fication criteria (43. 44 1. 

We select a total of 3735 lepton-|-jets data events with 
only one fo-tagged jet. The sample composition is given 
in Table H 

We further improve the sensitivity to FCNC through 
an application of Bayesian neural networks (BNN) [4J- 
li^ l. with settings identical to those detailed in Ref. [1[. 
A BNN is an average over many individual neural 
networks (100 networks are used in this analysis), 
where the parameters for each network are sampled from 



TABLE I: Event yields with uncertainty for each jet 
multiplicity for the electron and muon channels combined. 
The FCNC signals are each normalized to their observed cross 
section upper limits. The uncertainty on the total background 
includes correlations amongst sources. 



Source 


2 jets 


3 jets 


4 jets 


FCNC signal 








tgu 


34 ±4 


16 ±3 


5±1 


tgc 


54 ±7 


23 ±4 


7±2 


Background 








W-l-jets 


1660 ± 146 


560 ± 54 


154 ±15 


Z+]ets and dibosons 204 ± 34 


72 ±14 


22 ±6 


SM single top 


112 ± 15 


46 ±7 


14 ±3 


tt 


152 ± 24 


277 ± 42 


278 ±41 


Multijets 


184 ± 47 


66 ±15 


27±5 


Total background 


2312 ± 170 


1021 ±84 


495 ± 53 


Data 


2277 


958 


500 



the Bayesian posterior density distribution of the entire 
network parameter space. 

We use 54 discriminating variables, a subset of 
those used in each channel of the single top quark 
observation anal ysis [l| plus those from the previous 
FCNC analysis [l8|. The set of variables comprises 
individual object and event kinematics, top quark 
reconstruction, jet width, and angular correlations. 
Figure [5] compares the observed data to the background 
model for six illustrative discriminating variables. Object 
kinematics, such as the leading jet pT, and event 
kinematics, such as the invariant mass of the all-jets 
system, help separate the FCNC signals from the VF+jets 
background. Jet reconstruction variables, such as the 
width in 77 of the second leading jet, provide additional 
separation of light quark jets and heavy flavor jets. 
Angular variables such as the cosine between lepton 
and leading jet, or the 4> difference between lepton and 
^T, separate FGNG interactions from all backgrounds. 
Reconstruction of the top quark by combining the 
W boson with one of the jets discriminates against the 
IF+jets background. The top quark mass reconstructed 
with the leading jet separates FCNC signal events (where 
the leading jet typically comes from the top quark decay) 
from all backgrounds including tt (where the leading jet 
comes from one of the two top quark decays). 

Since their kinematics are similar, the two FCNC 
processes are combined into a single signal for training 
the BNN, each normalized to the same coupling. 
Separate BNNs are trained for each choice of lepton 
flavor (electron or muon), jet multiplicity (2, 3, or 4), 
and data-taking period, twelve in total. Each utilizes 
23 or 24 variables, selected from the list of 54, to 
provide the highest sensitivity for each analysis channel. 
Figure [3] shows the comparison between background and 
data for all twelve BNN discriminants combined. A 
Kolmogorov-Smirnov test comparing the observed data 
to the background sum in Figs.|3Ja) and[31Jb) gives values 
of 0.38 and 1.0, respectively, demonstrating that the 
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FIG. 2: Comparison of the background model to data for 
several discriminating variables summed over all analysis 
channels: (a) pr of the leading jet, (b) invariant mass of 
the system of all jets, (c) width in pseudorapidity of the 
second leading jet, (d) cosine of the angle between the leading 
jet and the lepton, (e) <j) separation between the lepton 
and ^T, and (f) top quark mass reconstructed from the 
reconstructed W boson and the leading jet. The FCNC 
signals are normalized to cross sections of 5 pb to visualize 
them clearly, and W/Z + X includes VK+jets and smaller 
backgrounds from Z+jets and dibosons. 



background model reproduces the data well. 
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FIG. 3: Comparison of the background model to data for 
the FCNC discriminant summed over all analysis channels, 
(a) for the whole discriminant range and (b) only the high 
discriminant region, where the hatched region gives the 
uncertainty on the background sum. The bins have been 
ordered by their signal to background ratio and the FCNC 
signals are each normalized to a cross section of 5 pb in (a) 
and to their observed limits in (b). W/Z + X includes W+jets 
and smaller backgrounds from Z+]ets and dibosons. 



Systematic uncertainties on the modeling of signal 
and background are described in Ref. 4J], with main 
uncertainties being from corrections to the jet energy 
scale and the &-tag modeling, with smaller contri- 
butions arising from MC statistics, corrections for jet- 
flavor composition in M^-|-jets events, and from the 
normalization of VF-l-jets, multijets, and tt production. 
The total uncertainty on the background is (8-16)%, 
depending on the analysis channel. For jet energy scale, 
6-tag modeling and VF-|-jets modeling, we vary not only 
the normalization but also consider effects on the shape of 
the final discriminant. When setting limits on the FCNC 
couplings, an additional signal cross section uncertainty 
of 8.8% from the NLO calculation is included 24 1. 

Since the data are consistent with the background 
expectation, we set upper limits on the FCNC cross 
sections and couplings using a Bayesian approach 48|. 
Following the analysis strategy of our previous work 18| , 
we form a two-dimensional Bayesian posterior density 
for the cross sections and for the square of the 
FCNC couplings, using the BNN distributions for data, 
background, and signals. Systematic uncertainties are 
taken into account with Gaussian priors, including 
correlations among bins and signal and background 
sources. We choose priors that are flat and non-negative 
in the FCNC couplings squared and hence in the FCNC 
cross sections. The posterior density as a function of the 
FCNC cross sections atgu and atgc is shown in Fig. |H[a) . 
We similarly form a two-dimensional Bayesian posterior 
density as a function of the (Kjg^/A)^, as shown in 
Fig. mjb), adding systematic uncertainties to the FCNC 
cross sections. 

One-dimensional posterior densities as a function 
of atgu and atgc are derived from the general two- 
dimensional posterior, by integrating over the atgc or 
atgu axes, respectively. One-dimensional posteriors are 
similarly derived as a function of {ntgu/ ^Y' and {ntgd 
and are shown in Fig. [S] This procedure keeps the 
measurement free of theoretical assumptions concerning 
the relationship between the two FCNC cross sections 
and couplings. For each quantity, we also compute 
expected limits by replacing the count in data in each 
bin by the background sum. The expected posterior 
densities for (K(g„/A)^ and (ntgc/A)'^ are also shown 
in Fig. [51 together with the 95% C.L. limits. The 
observed limits are below the expected limits, consistent 
with Fig. ^h), which shows that the data count is 
below the background expectation for several bins in 
the high BNN output region. Since the FCNC decay 
branching fraction is proportional to the square of the 
coupling, the limits on the couplings can be translated 
into decay branching fraction limits based on the NLO 
calculation [25|. The limits on cross sections, couplings 
and branching fractions are summarized in Table [III 

In summary, we have presented a search for FCNC 
interactions in the gluon coupling of top quarks to 
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couplings. 



TABLE II: Observed 95% C.L. upper one-dimensional limits 
on the FCNC cross sections, couplings, and branching 
fractions. 





tgu 


tgc 


Cross section 


0.20 pb 


0.27 pb 


Ktgf/A 


0.013 TeV"^ 


0.057 TeV~^ 


Bit ^ fg) 


2.0 X 10"* 


3.9 X lO"-'' 



up quarks or charm quarks. Using a sample of 2.3 fb"^ 
of integrated luminosity recorded by the DO experiment 
at the Tevatron Collider at Fermilab, we set limits on 
the couplings of Kfg„/A < 0.013 TeV~^ and Ktgc/A < 
0.057 TeV~^, without making assumptions about the tgc 
and tgu couplings, respectively. The corresponding limits 
on top quark decay branching fractions are B{t ^ gu) < 
2.0 X 10""* and B{t -> gc) < 3.9 x 10^^. These branching 
fraction limits are the most stringent and improve on the 
previous best limits by factors of two for B(i — > gu) and 



1.5 for B{t — 7> gc) [2l|. They improve on DO's previous 
result by a factor eight as a result of a larg er data set 
and significant improvements in analysis jl8l |. 
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